1. Field of the Invention
The present invention relates to an image processing apparatus for picking up (selecting) a motion picture and recording or transmitting the picked up motion picture, and more specifically, to an image processing apparatus having an automatic image pickup (selecting) feature that is suitable, for instance, for a monitor system and for observations of animal and plant ecology and the like.
2. Description of the Background Art
In recent years, progress in digitization of image equipment is being made due to progress and advances made in the digital signal processing techniques and digital coding techniques for images. In the field of image recording apparatuses, digital video cameras, digital still cameras, and the like are becoming widespread, and in such equipment, an input image signal is first digitized, and image signals are all handled as digital information throughout the processing, from signal processing such as coding to a process of recording onto a medium.
In this manner, when digital signal processing is more commonly performed within the image equipment, it becomes easier to implement a variety of features by additionally providing a signal processing feature to such equipment on top of other features required of such equipment to serve its originally intended purposes such as coding and conversion into a recording format.
A representative example of such additional features is an automatic image pickup feature used in a monitor system for security supervision, observation of animal and plant ecology, and so on. In the case of security supervision, such feature involves permanent provision of an image pickup unit such as a camera to be directed toward a target to be monitored, detecting a change in image content due to appearance of an intruder or occurrence of abnormality, and recording or transmitting to a central managing unit and the like the images from the time a change is detected so as to allow use of the images in notification of abnormality, in identification of the intruder, and in assessment and confirmation of the situation as well as to achieve saving of storage capacity and cutting back in transmission cost.
Moreover, in an example of ecology observation, such feature are realized as recording and transmitting only the images in which a target animal makes its appearance, and recording and transmitting at intervals of a few seconds or few tens of seconds the images representing a phenomenon that undergoes a slow change such as blooming of a flower or an ecdysis of an insect.
In the application for a monitor system as described above, a user is unable to operate the equipment constantly so that the essential techniques include a technique of detecting a change in the image content from input image signals and a technique of automatically controlling recording and transmission of images according to the detected result.
Images, particularly motion pictures, contain a very large amount of data so that normally upon recording and transmission, the amount of information is reduced using coding. In general, in coding of a motion picture, inter-frame or inter-field prediction is employed to extract and utilize only the information in which a change was found between picture planes.
Thus, for a certain picture plane, only the differential information from a previous picture plane that is referenced upon prediction would be recorded and transmitted as coded data. Upon prediction between picture planes, a process is also used which involves detecting a motion from a prediction reference picture plane per local region, local regions being obtained by dividing a picture plane into such local regions, and compensating for such motion.
Such process is called motion compensation. In addition, a motion in a local region can be represented as a two-dimensional vector value consisting of a value of horizontal component and a value of vertical component of the motion so that it is referred to as a motion vector.
In this manner, in the coding processing of a motion picture, a component representing a change in a picture plane is extracted from an input image signal and becomes an object of processing. Since such technique is the equivalent of the detection of a change in images in the above-described monitor camera system, a monitor camera system is realized which utilizes the processing in motion picture coding by itself or in combination.
Conventional examples of such monitor camera system that are known include devices described in Japanese Patent Laying-Open Nos. 6-153199, 10-285542, 10-322682, and 11-18076, for instance. These conventional techniques will be described below.
Japanese Patent Laying-Open No. 6-153199 proposes the monitor equipment in which a state change is detected in state change detection means based on input picture information or on compressed picture information having undergone compression processing, and when a change is detected, transmission means is activated in order to transmit the compressed picture information from before, in the midst of, and after the state change to a monitor station and the like.
Moreover, the above Japanese Patent Laying-Open No. 6-153199 discloses using an infrared sensor or a laser beam as the state change detection means to sense an intrusion of a person as well as detecting a state change based on a difference between the latest compressed image and a reference compressed image or based on a difference per pixel between an input image before compression and a reference image.
In addition, a method in which, instead of employing the simple difference per pixel, an image is divided into blocks and luminance values of all pixels in a block are accumulated for each of the blocks so as to allow detection of presence or absence of an object based on an amount of change in the cumulative values between images is disclosed, for instance, in Japanese Patent Laying-Open No. 9-251541.
Japanese Patent Laying-Open No. 10-285542 proposes an image recorder which stores in a recording medium prescribed images among input images, makes a comparison between input images and prescribed images using comparing means, and controls starting and stopping of image recording of input images according to a result of the comparison.
Moreover, the above Japanese Patent Laying-Open No. 10-285542 discloses detecting a motion vector in the input images using prescribed images as prediction reference images to compare a threshold value with a residual after motion compensation according to the motion vector using the comparing means, and allowing shared use of a processing portion in image compression coding means for the motion vector detection and the motion compensation.
In addition, a method of detecting a difference in total activity or per-block activities of image frames obtained from an operation result by discrete cosine transform upon compression coding processing so as to detect a motion in a video image or a motion vector is disclosed, for instance, in Japanese Patent Laying-Open No. 6-165181.
Japanese Patent Laying-Open No. 10-322682 proposes a monitor device for remote moving pictures in which a motion vector is detected in motion vector detection means between an input moving picture and an image of a previous frame, the input moving pictures are encoded by a moving picture encoding part based on the motion vector, a moving object detection part determines that a moving object is detected when a rate of blocks whose norm (magnitude) of motion vector is greater than a threshold value is within a prescribed range, and upon such determination, an ISDN (Integrated Services Digital Network) line connection is established to transmit the input images.
Japanese Patent Laying-Open No. 11-18076 proposes a video recorder having a time-lapse recording feature that enables intermittent recording of video signals at constant intervals, wherein input video signals are coded using MPEG (Moving Picture Experts Group) image compression in a video data converting part, and when a motion of video data is detected from the MPEG image compression information, the video signals from that time zone is precisely recorded as continuous dynamic images, and not through time-lapse recording.
In the conventional monitor systems described above, detection and notification of an intruder and occurrence of abnormality is the main purpose so that the systems are configured such that only those images in which a state change is detected as a result of comparison with a reference image or an initial input image are recorded and transmitted.
In applications such as a security supervision system, the target to be monitored is normally specified so that the content of the reference image is already known, and such arrangement as described above is sufficient, since no significance is found in recording and transmitting images that contain no changes.
When recording only the images in which a state change is detected, however, the recorded images become awkward representations that capture only the changes. Moreover, there is a problem of no recording taking place when the movement of an intruder is small and thus the change in the images is small.
Furthermore, there is a need to provide a special control that prevents recording of images and additional information when no change takes place in the images from the activation of an automatic image pickup feature to the termination of the automatic image pickup.
In addition, as described in Japanese Patent Laying-Open Nos. 6-153199 and 10-322682, transmission means is activated according to a state change in images so that, processes of connection and disconnection of a communication line must be performed frequently where a large state change and a small state change are alternately repeated. As a result, not only does the processing become burdensome but also negotiation delay, initial transmission delay, and the like occur in the connection procedure every time connection is established.
Moreover, in the case where image information from before, in the midst of, and after the detection of a state change, particularly, the images from the time before the change are to be transmitted, as described in Japanese Patent Laying-Open No. 6-153199, there is a need to provide a storage unit for retaining input or coded image information in preparation for an occurrence of a change at all times. Moreover, there is a problem of a significant delay after the detection of the state change due to starting the transmission from previous image information.
In the above Japanese Patent Laying-Open No. 6-153199, the images of a prescribed period that extends before and after the time of state change detection are merely handled as one mass. Thus, after a change takes place, if another change in images takes place during transmission related to the previous change, the latter change would not be detected, or even if it were detected, additional image information corresponding to the latter change cannot be transmitted, since the device is already in the midst of a transmission.
Moreover, with regard to a method of detecting a state change in images, when a difference of pixel values between images are utilized as described in Japanese Patent Laying-Open Nos. 6-153199 and 9-251541, while successful detection of any change can be ensured, such changes as a noise contained in an input image, changes in illumination and brightness, the entire image of a background or a scenery moving slightly, and so on whose recording is desirably to be avoided are also detected as changes.
Furthermore, in the case where a prediction error from after motion compensation prediction upon compression coding processing of images is employed in detection of a change as described in Japanese Patent Laying-Open No. 10-285542, the magnitude of the prediction error does not necessary represent the degree of change contained in the images, since the prediction error becomes small if the motion compensation prediction works effectively even when the images contain a motion.
On the other hand, in the case where a change in images is detected using a motion vector as described in Japanese Patent Laying-Open No. 10-322682, the motion vector being utilized is one that was detected for the purpose of image coding processing so that it does not necessarily express an accurate motion in the images. In addition, complex changes of images and a motion accompanying variation in color and brightness cannot be detected as motions so that they cannot be determined as state changes of images.
Moreover, when an apparatus automatically controls the starting and stopping of recording and transmission of images as described above, it becomes difficult for a user to determine the operative condition of the apparatus from outside, i. e. whether the apparatus is currently recording, transmitting, or being suspended. Thus, conventionally, characters indicating the operative condition or a marker and the like are superimposed for display on a display portion of the input image such as a viewfinder or a monitor.
Normally, in applications for monitor systems and the like, however, an image recording apparatus or a transmission apparatus is permanently provided by being directed toward a target to be monitored, while the user is at a remote location. In other words, the user is never observing the viewfinder, the monitor, or the like so that the user cannot determine the operative condition of the apparatus even when the condition is being represented on the viewfinder or the monitor. Thus, the user is unable to identify the operative condition of the apparatus from a remote location.